fix(photon): transcode inbound HEIC images - #58934
samsungapore wants to merge 1 commit into
Conversation
e9bdc7a to
b0efbe0
Compare
alt-glitch
added
type/bug
Related: #54946 (broader canonical HEIC/HEIF fix — centralizes detection in the shared |
teknium1
left a comment
teknium1
left a comment
There was a problem hiding this comment.
Thanks for tracing the Photon path and adding a native-vision regression test. The current main path does have the HEIC failure: plugins/platforms/photon/adapter.py:1637-1644 falls back to a document after the shared image cache rejects HEIC magic, while retaining the stale image MIME.
Problems
plugins/platforms/photon/adapter.py:1560calls blockingsubprocess.run()from the async inbound dispatch path. A slowsipsinvocation can block Photon event-loop progress for up to the configured 20 seconds.plugins/platforms/photon/adapter.py:1613-1616returnsNonewhen conversion is unavailable or fails. That causes_normalize_binary_payloadto surface only a marker, dropping otherwise available attachment bytes instead of preserving them as a document.- The member comment correctly identifies open #54946 as the broader shared-cache approach. It covers Photon and BlueBubbles and includes a document fallback, which avoids duplicating HEIC policy in the Photon adapter.
Suggested changes
- Rework this onto the shared cache path, with conversion offloaded from the event loop and a document fallback on conversion failure.
- Add a failure/unavailable-converter regression test alongside the successful JPEG test.
Automated hermes-sweeper review.
| src = Path(tmp) / f"source{safe_suffix}" | ||
| dst = Path(tmp) / "converted.jpg" | ||
| src.write_bytes(raw) | ||
| proc = subprocess.run( # noqa: S603 - fixed executable + temp paths |
There was a problem hiding this comment.
_transcode_heif_to_jpeg() is reached synchronously from async _dispatch_inbound, so this subprocess.run() blocks Photon event-loop progress for up to 20 seconds. Please offload conversion through an async shared-cache path (for example, asyncio.to_thread) rather than blocking inbound delivery.
| if _is_heif_image(mime, suffix): | ||
| converted = _transcode_heif_to_jpeg(raw, suffix) | ||
| if converted is None: | ||
| return None |
There was a problem hiding this comment.
Returning None here drops attachment bytes and makes _normalize_binary_payload emit only a metadata marker. Preserve the original HEIC as a document with a non-image MIME when conversion is unavailable or fails; this keeps the attachment inspectable without routing unsupported bytes to native vision.
teknium1
added
the
sweeper:blast-moderate
|
I reproduced the generic inbound HEIC attachment path. The image must be transcoded before downstream processing. The fix preserves the existing platform and cluster mapping and uses a synthetic fixture with no production-specific values or captured filenames. |
This branch has not been deployed
4 participants
Summary
sipsimage/jpeg) intoMessageEvent.media_typesafter conversionRoot cause
Inbound iPhone photos can arrive as HEIC/HEIF bytes. Photon cached them with stale
image/heicmetadata, so native image routing attempted to attach an unreadable HEIC path whenpillow-heifwas not installed. The model then saw only the iMessage placeholder/attachment marker instead of the image.Test plan
scripts/run_tests.sh tests/plugins/platforms/photon/test_inbound.py tests/agent/test_image_routing.py -q→ 111 passed./venv/bin/python -m py_compile plugins/platforms/photon/adapter.pynode --check plugins/platforms/photon/sidecar/index.mjsgit diff --checkLive verification
Verified locally with a real inbound iMessage photo: Photon cached a fresh
img_*.jpgwith JPEG magicffd8ff, gateway logged native image routing with 1 image attached inline, and Photon sent the model response back successfully.